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SUMMARY 


This  is  a  general  description  of  an  automated  and  intelligent 
tool  to  assist  courseware  authors  in  instructional  design  and 
development.  The  problem  addressed  by  this  research  is  the 
difficulty  and  expense  of  designing  and  developing  effective 
courseware  given  the  complexities  of  advanced  hardware  and 
software  technologies,  the  variety  of  instructional  settings, 
and  the  relative  inexperience  of  Air  Force  training 
specialists  in  instructional  design.  Many  automated  tools  to 
support  the  instructional  process  are  being  developed. 
However,  no  existing  system  attempts  to  provide  general 
guidance  for  effective  courseware  design  and  development.  The 
Advanced  Instructional  Design  Advisor  (AIDA)  is  currently 
being  designed  and  developed  by  AFHRL/IDC  to  explore  the 
feasibility  of  providing  automated  and  intelligent  assistance 
to  Air  Force  training  specialists  involved  in  courseware 
development.  ^  In  order  to  design  an  effective  set  of  tools, 
several  critical  issues  need  to  be  resolved,  including  the 
identification  of  useful  instructional  models  and  taxonomies 
of  knowledge,  a  determination  of  the  role  for  artificial 
intelligence  in  instructional  design,  and  an  analysis  of 
instructional  design  requirements.  This  research  effort 
represents  a  first  attempt  to  address  these  critical  design 
issues  for  AIDA. 
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PREFACE 


In  the  summer  of  1988,  the  Air  Training  Command  issued  a 
Manpower,  Personnel,  and  Training  Need  (MPTN  89-14)  citing  a 
lack  of  effective  CBI  (computer-based  instruction)  deployment 
and  proposing  the  development  of  specifications  and  guidelines 
for  effective  authoring  and  presentation  complete  with 
instructional  strategies.  The  Advanced  Instructional  Design 
Advisor  (AIDA)  is  an  exploratory  R&D  effort  by  AFHRL/IDC  in 
response  to  this  MPTN.  The  current  AIDA  design  effort  is 
being  done  by  Mei  Associates,  Inc.,  Lexington,  MA  (Contract 
Number  F33615-88-C-0003) . 

The  preliminary  conceptual  work  on  AIDA  was  done  as  part  of 
my  Air  Force  Summer  Faculty  Research  Program  grant  at 
AFHRL/IDC  in  1988  (Contract  Number  F49620-87-R-0004)  and 
Research  Initiation  Program  grant  at  AFHRL/IDC  in  1989 
(Contract  Number  F49620-88-C-0053) .  This  report  is  primarily 
a  synthesis  of  the  findings  of  those  two  research  efforts. 

I  would  like  to  thank  the  United  States  Air  Force  Systems 
Command,  the  Air  Force  Office  of  Scientific  Research,  and 
Jacksonville  State  University  for  sponsorship  of  this 
research.  Universal  Energy  Systems  was  very  helpful  in 
providing  administrative  support.  I  am  grateful  to  the  Air 
Force  Human  Resources  Laboratory  for  providing  a  stimulating 
environment  for  my  research. 

The  work  that  has  been  accomplished  on  the  Advanced 
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Instructional  Design  Advisor  is  the  result  of  an  unusually 
productive  working  relationship  that  has  developed  between  the 
academic  and  military  consultants  working  with  Mei  Associates, 
Inc.  on  this  project.  I  value  highly  the  many  and  significant 
contributions  of  Drs.  John  Ellis,  Alinda  Friedman,  Robert 
Gagne,  Henry  Halff,  Albert  Hickey,  Eileen  Kintsch,  Mary 
Marline,  David  Merrill,  Harry  O'Neil,  Martha  Poison,  Charles 
Reigeluth,  Robert  Seidel,  Robert  Tennyson,  Lt.  Cols.  Hector 
Acosta,  Jerry  Barucky,  Mike  Bush,  Larry  Clemons,  Mike 
Dickinson,  Dan  Meigs,  Richard  Ranker,  Ma j  .  Bob  Mongillo, 
Capts.  Ed  Arnold,  Jim  Coward,  Woollard  Rosamond,  and  Mr.  Brian 
Dallman. 

I  especially  appreciate  the  thoughtful  guidance  of  Dr.  Scott 
Newcomb,  AFHRL/IDC  Branch  Chief.  His  ideas  and  suggestions 
have  been  most  beneficial.  In  addition,  I  appreciate  the  help 
and  support  of  the  other  personnel  in  the  Training  Technology 
Branch  (AFHRL/IDC) ,  including  Major  Bill  Wimpee,  Dr.  Linda 
Curran,  Dr.  Dan  Muraida,  Mr.  Dick  Vigue  (retired),  Mrs. 
Barbara  Eaton  (transferred) ,  Capt.  Bill  Dimitroff ,  Mr.  Bill 
Hawks,  Mr.  Dennis  Gettman,  and  Ms.  Catherine  Connolly.  Mrs. 
Sherri  Barnes  provided  administrative  assistance  well  above 
and  beyond  the  call  of  duty.  Other  AFHRL  scientists  offered 
many  useful  ideas,  especially  Lt.  Col.  Hugh  Burns  (retired) 
and  Drs.  Barbara  Sorensen,  Wes  Regian  and  Kurt  Steuck.  I  also 
appreciate  the  encouragement  and  support  offered  by  Dr.  Henk 
Ruck  and  Col.  Rodger  Ballentine. 
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I .  INTRODUCTION 

Phases  of  Instruction 

The  instructional  process  involves  three  major  phases:  1) 
front-end  analysis  (FEA) ,  2)  design,  development,  and  delivery 
(DDD) ,  and  3)  rear-end  analysis  (REA) .  Instructional  Systems 
Design  (ISD)  has  been  used  by  the  Air  Force  to  guide  progress 
in  all  three  phases,  although  its  usefulness  is  somewhat 
limited  since  it  provides  insufficient  specific  and  detailed 
guidance  with  regard  to  courseware  design,  development,  and 
delivery.  ISD  does  provide  a  general  guideline  of  steps  to 
follow  in  designing  and  developing  instruction,  and  this 
general  guideline  will  most  probably  be  retained  in  some  form 
in  whatever  advanced  instructional  design  systems  are 
developed. 

Phase  1,  FEA,  includes  an  analysis  of  instructional 
requirements  and  the  design  requirements  for  the  instructional 
solutions,  including  requirements  for  various  courses.  Phase 
2,  DDD,  is  where  the  primary  course-level  and  lesson-level 
planning,  development,  and  implementation  occur.  The  work 
done  in  DDD  is  intended  to  comply  with  the  instructional 
requirements  of  FEA.  Phase  3,  REA,  is  basically  an  analysis 
of  how  well  the  results  of  DDD  met  the  requirements  of  FEA. 
The  boundaries  between  these  phases  are  not  always  sharply 
drawn,  and  the  phases  do  not  always  occur  in  a  linear  or 
chronological  order.  Sequences  are  usually  cyclic  and 
somewhat  irregular. 
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CBI  Problems 


These  three  phases  roughly  correspond  to  the  five  steps  in  the 
classic  waterfall  life-cycle  model  of  software  development: 
analysis,  design,  implementation,  test,  and  maintenance 
(Fairley,  1985)  .  As  is  the  case  with  many  software 
development  efforts,  the  first  and  third  phases  (FEA  and  REA) 
receive  minimal  attention  in  the  instructional  design  process. 
It  is  an  all  too  common  problem  to  become  absorbed  with  the 
details  of  ODD  and  to  short-circuit  both  FEA  and  REA.  A 
rational  and  balanced  approach  distributes  the  effort  among 
all  three  phases.  One  of  the  long  range  goals  of  AIDA  is  to 
provide  a  full  range  of  automated  and  intelligent  tools  to 
assist  in  all  major  phases  of  ISD.  The  near  term  AIDA  effort 
is  focused  on  DDD  with  regard  to  computer-based  instruction 
(CBI)  . 

As  advances  in  both  hardware  and  software  technologies  have 
occurred,  courseware  designers  have  naturally  wanted  to 
incorporate  a  wider  range  of  materials,  including  graphics, 
video,  and  sound,  in  a  much  more  complex  computer  environment. 
As  a  consequence,  many  persons  with  many  different  talents  and 
skills  are  involved  in  the  ISD  process,  especially  in  the  DDD 
phase.  For  example,  if  we  imagine  an  instructional  design 
effort  for  a  CBI  system,  we  immediately  see  a  need  for  subject 
matter  experts  (SMEs) ,  instructional  designers,  production 
specialists  (e.g.,  video  specialists),  and  system  specialists 
(i.e.,  those  knowledgeable  about  the  particular  computer 
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hardware  and  software  systems  being  used) . 


As  the  number  of  specialists  increases,  it  is  reasonable  to 
expect  course  development  time,  costs,  and  quality  to 
increase.  As  the  complexity  of  the  development  and  delivery 
systems  increases,  we  again  have  similar  expectations  about 
development  time,  costs,  and  quality. 

In  the  typical  Air  Force  instructional  setting,  these 
specialists  are  rarely  available.  The  problem  motivating  AIDA 
is  to  make  the  kind  of  expertise  such  specialists  bring  to 
courseware  design  and  development  available  and  accessible  to 
SMEs  in  an  on-line  computer  environment.  A  major  challenge  in 
designing  AIDA  is  to  design  a  system  that  reduces  course 
development  time  and  costs  and  yet  contributes  to  course 
quality  and  effectiveness  (see  Appendix) . 

Systematic  Solutions 

The  argument  presented  here  is  that  adopting  a  systematic 
approach  to  integrating  existing  theories  of  instruction, 
learning,  and  knowledge  in  an  easily  accessible  automated 
environment  will  provide  the  means  to  determine  which 
particular  instructional  prescriptions  are  appropriate  in  a 
variety  of  automated  settings .  As  AIDA  related  research 
proceeds,  these  validated  instructional  prescriptions  will 
form  the  central  component  of  an  automated  instructional 
design  advisor. 
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II.  OBJECTIVES  OF  THE  RESEARCH 


Purpose  &  Questions 

The  primary  purpose  of  this  research  effort  was  to  determine 
how  advances  in  the  cognitive,  instructional,  and  computer 
sciences  could  be  applied  to  the  design  of  an  advanced 
instructional  design  system-  In  order  to  achieve  this  goal, 
the  following  related  questions  were  addressed: 

1)  Which  theories  of  knowledge  might  be  useful  in  the 
instructional  design  process? 

2)  Which  learning  theories  and  taxonomies  might  be  useful? 

3)  Which  instructional  design  theories  and  models  might  be 
useful? 

4)  What  is  the  potential  role  of  AI  in  instructional 
design? 

5)  How  is  ISD  useful  in  the  process  of  instructional  design 
and  how  can  its  use  be  automated  and  improved? 

6)  Which  strategies  should  be  followed  in  order  to  design 
and  develop  advanced  authoring  aids? 

7)  What  kinds  of  advanced  authoring  aids  al.-eady  exist  and 
how  effective  are  they? 

In  formulating  answers  to  these  questions  it  became  clear  that 
a  standard  design  philosophy  to  guide  the  development  of  all 
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tools  built  to  support  the  instructional  process  was 
desirable.  It  was  also  clear  that  an  expert  syster  to  support 
instructional  design  was  possible. 

Sections  VIII  through  X  of  this  paper  present  initial  answers 
to  these  seven  questions.  Question  #5  about  the  usefulness  of 
the  existing  ISD  procedures  received  minimal  attention.  The 
assumption  here  is  that  R&D  efforts  should  not  be  constrained 
by  existing  ISD  procedures.  More  strongly,  it  would  be  a 
mistake  to  take  existing  ISD  procedures  for  classroom-based 
instruction  and  automate  them  for  CBT  settings  without  first 
establishing  their  appropriateness  for  CBT.  In  order  to 
establish  what  procedures  are  appropriate  for  designing  and 
developing  CBT,  one  should  perform  a  cognitive  analysis  of 
tasks  involved  in  designing  and  developing  CBT.  Such  an 
effort  was  clearly  beyond  the  scope  of  this  research. 

Specific  Tasks 

A  systematic  approach  to  answering  these  seven  question 
resulted  in  the  following  specific  tasks: 

1)  Evaluation  of  SOCRATES,  a  system  developed  and  used  at 
the  Air  University's  Air  Command  &  Staff  College  (questions 
4  and  5) . 

2)  A  description  of  useful  instructional  models  (questions 
1  and  5)  . 

3)  A  description  of  an  instructional  model  selection 
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algorithm  to  match  learning  objectives,  knowledge  types, 
instructional  settings,  and  instructional  strategies 
(questions  2  and  3) . 

4)  A  modular  description  of  AIDA  (questions  5  and  6) . 

5)  Participation  in  the  formal  review  of  the  design 
requirements  for  AIDA  (question  6) . 

6)  Evaluation  of  other  research  and  development  efforts  in 
this  area  (questions  4,  5,  and  7). 

7)  An  analysis  of  the  role  for  artificial  intelligence  in 
AIDA  (question  4) .  .Im  3 

Summary  of  Results 

SOCRATES  and  several  other  instructional  design  advising 
systems  were  evaluated.  Existing  systems  are  inadequate 
for  a  variety  of  reasons,  including  failure  to  account  for 
a  variety  of  instructional  settings  and  relatively 
inexperienced  instructional  designers. 

Useful  instructional  models  do  exist,  including  Gagne's 
Nine  Events  of  Instruction,  Merrill's  Component  Display 
Theory  (and  more  recent  Second  Generation  Instructional 
Design  Theory),  and  Reigeluth's  Elaboration  Theory. 
However,  these  theories  have  yet  to  be  incorporated  into  a 
relatively  general  purpose  automated  instructional  design 
system  (Merrill's  ID  Expert  probably  comes  the  closest). 
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No  existing  instructional  design  advising  system  was  found 
that  provided  genuinely  adaptive  instructional  models. 
ITSs  do  incorporate  adaptive  instructional  techniques,  but 
they  do  not  typically  adopt  a  completely  different 
instructional  model  as  instructional  circumstances  might 
dictate.  For  example,  an  expository  instructional  model 
Cr  teaching  a  procedure  might  propose  the  following: 

1.  Start  with  an  attention  grabber. 

2.  Present  a  motivational  scenario. 

3 .  Provide  a  reminder  that  this  procedure  is  similar  to  a 
familiar  procedure. 

4.  State  the  steps  comprising  the  procedure. 

5.  Provide  several  examples  of  following  the  procedure. 

6.  Provide  practice  and  assessment. 

If  it  is  determined  at  step  6  that  the  instruction  has  not 
been  effective,  then  several  alternatives  exist.  One 
alternative  is  to  retreat  to  step  4  (or  an  earlier  step)  and 
repeat  a  somewhat  abbreviated  sequence.  This  is  the  common 
tactic.  Another  alternative,  however,  is  to  adopt  an  entirely 
different  instructional  model.  It  is  not  inconceivable  that 
the  failure  is  due  to  a  breakdown  at  step  3,  the  analogy  step. 
To  attack  such  a  deficiency,  an  intelligent  system  might 
advise  (or  adopt  or  provide)  an  inquisitory  model  of 
instruction  that  proposes  these  steps; 

1.  State  the  purpose  of  the  procedure. 

2.  State  the  steps  of  the  procedure. 
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3.  Provide  several  examples  of  the  procedure. 

4.  Request  a  similar  procedure. 

5.  Request  similarities  and  differences. 

6.  Repeat  steps  3  and  4  several  times. 

7.  Provide  additional  examples  of  the  procedure. 

8.  Request  hypothesis  for  result  of  omitting  a  step. 

9.  Request  explanation  and  discussion. 

10.  Repeat  steps  8  &  9  several  times. 

11.  Provide  practice  and  assessment. 

Human  instructors  often  adapt  their  instruction  when 
circumstances  indicate  a  different  model  or  tactic  is 
appropriate.  An  intelligent  CBI  instructional  advisor  should 
have  a  similar  capability. 

There  clearly  is  a  role  for  artificial  intelligence  in  the 
instructional  design  and  development  process.  ITSs  have 
already  incorporated  expert  systems  to  provide  instruction 
based  on  matching  a  model  of  an  expert's  knowledge  with  that 
of  a  student's.  The  challenge  for  AIDA  is  to  provide  an 
expert  planning  model  for  designing  and  developing  CBI.  An 
additional  challenge  for  AIDA  and  for  ITSs  is  to  provide 
adaptive  instructional  models  in  the  instructional  delivery 
component . 

III.  PROBLEM  AREAS 

The  general  problem  with  DDD  is  that  advanced  hardware  and 
software  technologies,  including  simulations  using  interactive 
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video  disk  (IVD)  or  digital  video  interactive  (DVI) 
technology,  natural  language  pre-processors,  and  intelligent 
tutoring  systems  (ITSs),  have  found  roosts  in  various 
instructional  settings.  However,  our  knowledge  of  how  best  to 
use  such  tools  is  limited  (Montague,  Wulfeck,  &  Ellis,  1983) . 
These  technologies  generally  lack  standards,  are  relatively 
expensive,  require  expertise  to  use  effectively,  and  are  not 
always  placed  in  appropriate  settings.  The  basic  problem, 
however,  is  that  we  do  not  know  how  best  to  design  courses  so 
as  to  optimize  effectiveness,  given  that  broad  subject  matter 
domains,  a  variety  of  instructional  settings,  diverse  student 
populations,  and  new  technologies  are  involved  (Halff ,  1988) . 

A  review  of  the  literature  reveals  no  general  consensus  with 
regard  to  a  theory  of  instructional  design  (Gagne,  1987) . 
Some  studies  reveal  that  it  often  matters  not  whether  certain 
portions  of  a  course  are  omitted  or  whether  entirely  different 
approaches  are  used  to  teach  the  material  (Montague,  1988) . 

In  order  to  achieve  the  research  objectives  and  to  support  the 
Laboratory's  efforts  to  design  an  advanced  instructional 
design  toolset,  several  different  problems  were  evaluated.  In 
determining  the  current  state  of  instructional  design  science, 
two  problem  areas  were  explored:  1)  Can  theories  of 
knowledge,  learning,  and  instructional  design  be  meaningfully 
integrated?,  and  2)  What  theories  are  existing  instructional 
design  systems  incorporating?  (These  two  questions  represent 
questions  l,  2,  3,  7  and  tasks  1,  2,  3,  and  4  in  Section  II). 
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In  order  to  determine  the  role  for  artificial  intelligence  in 
AIDA,  again  two  problem  areas  were  explored:  1)  Is  there  a 
role  for  expert  systems  in  the  ISD  process?,  and  2)  Is  there  a 
role  for  neural  networks  in  the  ISD  process?  (These  two 
questions  represent  questions  4  and  5  and  tasks  6  and  7  in 
Section  II) . 

In  order  to  provide  support  for  the  Laboratory's  AIDA  effort 
under  Contract  Number  F33615-88-C-0003  with  Mei  Associates, 
Inc. ,  two  additional  problems  were  explored:  1)  What  kind  of 
consultants  should  be  involved  in  the  design  effort?,  and  2) 
What  specific  tasks  should  be  assigned  these  consultants? 
(These  two  questions  represent  question  6  and  task  5  in 
Section  II) . 

IV.  PROPOSED  SOLUTION 

The  proposed  solution  to  courseware  design  and  development 
problems  is  to  develop  a  computer-based  toolset  to  assist  in 
the  process.  The  specific  purpose  of  this  set  of  tools  is  to 
reduce  course  development  time  and  costs  while  assisting  in 
the  production  of  consistently  effective  courses.  The  cost 
savings  will  come  in  the  form  of  reduced  needs  for  expensive 
and  scarce  design  and  systems  expertise  as  well  as  in  time 
saved  in  course  development.  This  means  that  such  a  system 
must  be  inexpensive,  easy  to  learn,  and  obvious  to  use. 

In  attempting  to  design  a  system  to  improve  the  course  design 
process,  three  important  questions  will  eventually  need  to  be 
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resolved:  1)  What  are  the  most  promising  theories  of 
instruction?  2)  how  can  their  prescriptions  be  followed  and 
tested?  and  3)  What  are  the  limitations  of  implementations  of 
such  systems?  Sections  VIII  through  X  only  sketch  an  answer 
to  these  additional  questions. 

A  review  of  the  literature  and  visits  to  military 
instructional  research  and  development  centers  reveal  that 
there  are  no  existing  systems  that  claim  to  solve  this 
fundamental  problem  in  a  general  way.  The  Air  University  at 
Maxwell  AFB  has  developed  a  tool,  SOCRATES  (Doucet  and  Ranker, 
1988) ,  that  does  provide  lesson  assistance  according  to 
Gagne's  nine  events  of  instruction  and  Merrill's  Component 
Display  Theory  (Reigeluth,  1983). 

Merrill,  with  partial  funding  from  the  Army  Research  Institute 
(ARI)  designed  ID  Expert  Versions  1.0  &  2.0  (Merrill,  1986) 
which  provide  somewhat  similar  guidance  in  the  lesson 
construction  process. 

Such  tools  are  significant  steps  forward.  Their  limitations 
are  that  they  provide  restricted  guidance  for  highly 
constrained  settings.  The  few  existing  instructional  design 
aids  that  provide  design  guidance  typically  focus  on  subject 
matter  involving  declarative  knowledge  and  are  not  suitable 
for  maintenance  training  tasks.  AIDA  is  meant  to  be  a  more 
general  solution  at  the  course  level,  and  it  is  meant  to  fit 
within  the  larger  scheme  of  the  instructional  process. 
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V.  SOLUTION  CONTEXT 


Tools  for  FEA 

Before  describing  AIDA  in  more  detail,  it  is  useful  to 
consider  its  context  in  the  process  of  instructional  design. 
It  is  reasonable  to  expect  a  set  of  advanced  tools  to  be 
developed  to  automate  and  assist  throughout  the  instructional 
process.  To  complement  the  ISD  process  during  FEA,  several 
programs  might  be  useful.  One  might  be  a  Training 
Requirements  Analysis  Planner  that  would  take  a  roughly  stated 
training  requirement  and  via  a  query/response  process  develop 
that  statement  into  a  set  of  requirements  and  goals  that  are 
in  an  appropriate  format  to  guide  the  courseware  design 
process.  Another  automated  aid  might  be  a  system  to  query  a 
course  database  to  find  out  if  similar  courses  exist,  where 
they  are  taught,  etc.  Yet  another  could  be  used  to  guide  the 
media  selection  process.  The  FEA  programs  could  also  automate 
required  ISD  documentation. 

Tools  for  DDD 

In  the  DDD  phase  a  variety  of  tools  are  possible.  A  program 
might  be  developed  to  take  lesson  objectives,  content,  and 
definitions  as  input  and  generate  test  questions,  much  as 
CBESS  does  for  restricted  domains  (Wetzel  &  Wulfeck,  1987) . 
Entire  systems  can  be  built  to  allow  authors  to  select  and 
sequence  lesson  materials;  ISS  is  an  excellent  example  of  an 
Air  Force  owned  authoring  system  (Vigue,  1988) .  Merlin  is  a 
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second  example  of  an  Air  Force  owned  authoring  environment. 
Neither  of  these  systems  provides  instructional  design 
guidance.  Both  are  highly  sophisticated  and,  like  their 
commercial  counterparts,  are  difficult  to  learn. 

Other  tools  that  would  be  useful  are  IVD  and  graphics  editors 
and  databases  containing  sample  course  and  lesson  templates 
and  examples.  As  indicated  earlier,  what  is  especially  needed 
in  the  ODD  phase  is  a  tool  to  assist  in  course  design  by 
matching  instructional  options,  knowledge  types,  and  settings 
(Madni,  1986). 


Tools  for  REA 

It  is  also  possible  to  imagine  a  range  of  tools  associated 
with  REA,  including  guides  for  establishing  and  conducting 
cirucial  controlled  tests  and  access  to  databases  containing 
relevant  course  data.  A  particularly  difficult  task  in  this 
area  is  testing  the  effectiveness  of  a  course  design  process, 
as  opposed  to  the  effectiveness  of  a  particular  course. 

An  AIDA  Toolset 

In  the  long  range,  AIDA  is  concerned  with  providing  a  standard 
environment  (graphical  user  interface,  design  philosophy, 
etc.)  for  a  fully  integrated  set  of  tools  to  support  all 
phases  of  the  instructional  systems  development  process.  In 
the  near  term,  AIDA  should  focus  on  design,  development,  and 
delivery  of  maintenance  training  in  computer-based 
instructional  settings. 
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VI .  FUNDAMENTAL  CONCERNS 


Kinds  of  Tools 

Looking  at  the  kinds  of  advanced  technologies  which  might  be 
developed  to  assist  throughout  the  instructional  process 
reveals  two  distinctions: 

1)  There  are  two  different  kinds  of  tools  that  can  be 
developed  —  those  which  automate  an  established  process 
(e.g.,  documentation  and  test  generators)  AND  those  which 
are  intended  to  make  the  process  more  effective  (e.g., 
course  and  lesson  templates,  ITSs,  and  lesson  and  course 
design  advisors) 

2)  The  tools  that  have  been  described  can  be  used  in  two 
ways  —  as  stand-alone,  special-purpose  tools  (e.g., 
practice  problem  generation)  OR  as  integrated  tools  sharing 
information  (e.g.,  a  course  database). 

Standard  Design  Philosophy 

The  guiding  vision  of  AIDA  is  to  adopt  a  standard  design 
philosophy  for  all  related  tools  so  that  as  tools  and  mini¬ 
advisors  are  developed  they  can  be  gracefully  incorporated 
into  a  system  of  tools  for  the  entire  instructional  process. 
Part  of  the  standard  design  philosophy  should  be  this 
fundamental  modular  concept.  Tools  should  be  developed  so 
that  they  can  operate  as  stand-alone  modules  or  can,  by  way  of 
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standard  interfaces,  be  incorporated  into  larger  collections 
of  compatible  modules.  A  standard  design  philosophy  is  needed 
to  guide  specific  items,  such  as  menu  design,  as  well  as 
larger  concerns,  such  as  the  structure  of  knowledge  bases. 
Standards  for  networking  and  graphics  are  also  needed. 

AI-Based  Tools 

Some  tools  developed  to  make  course  design  more  effective  fall 
into  the  area  of  AI,  such  as  ITSs  (Wenger,  1987)  ,  while  some 
fall  more  clearly  into  the  traditional  disciplines  of  computer 
science,  such  as  database  design  and  implementation.  AIDA  has 
some  aspects  of  both  kinds  of  systems.  A  serious  question  is 
the  extent  that  course  design  lends  itself  to  AI  applications 
(Winograd,  1986) .  Can  an  expert  system  for  course  design  be 
developed  that  performs  as  effectively  as  good  human 
designers? 

The  initial  assumption  is  that  AI  has  a  role  to  play  in  course 
design,  as  well  as  its  already  recognized  role  in  ITSs  (Poison 
&  Richardson,  1988) .  The  form  of  such  systems  will  be  sets  of 
production  rules  and  inference  engines  to  take  the  user  from 
one  state  (e.g.,  specific  list  of  course  objectives  and 
knowledge  of  the  expected  course  environment)  to  another  state 
(e.g.,  set  of  lesson  templates  outlining  the  course  content 
and  sequencing) . 
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Theories  of  Knowledge 


The  AI  questions  suggest  other  concerns:  1)  Are  there 
existing  theories  of  instruction  design  that  could  be 
incorporated  as  models  in  an  expert  system?  2)  Do  guidelines 
exist  to  suggest  which  models  to  apply  in  particular 
circumstances?  and  3)  What  relationships  exist  between 
theories  of  knowledge,  learning,  and  instruction? 

The  view  here  is  that  relationships  between  theories  of 
knowledge,  learning,  and  instruction  should  be  expected,  with 
theories  of  knowledge  providing  links  between  theories  of 
instruction  and  learning  (Self,  1988).  The  theory  of 
knowledge  assumed  below  is  a  modification  to  the  justified 
true  belief  model  discussed  in  Griffiths  (1967)  and  adopted  by 
Kitcher  (1983).  The  pragmatic  modification  is  this; 
Knowledge  consists  of  well- j ustif ied  and  well-connected 
beliefs  (JCB) .  What  counts  as  "well"  in  either  case  depends 
on  the  particular  circumstances  and  needs  at  hand. 

In  addition,  JCB  theory  acknowledges  the  usefulness  of  the 
distinctions  between  declarative  (knowing  that) ,  procedural 
(knowing  how) ,  and  causal  knowledge  (knowing  why) .  Each 
knowledge  type  has  possible  subdivisions  (e.g.,  the 
distinction  between  physical  process  and  mental  process  within 
procedural  knowledge) .  Human  knowledge  is  like  a  web  of  well- 
justified  and  well-connected  beliefs  (Quine  &  Ullian,  1978) . 
There  are  too  many  threads  to  explore  individually  in  order  to 
precisely  model  the  human  belief  system.  However,  useful 
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models  can  be  built  by  emphasizing  the  incorporation  and 
integration  of  multiple  knowledge  types,  according  to  the  most 
appropriate  learning  and  instruction  theories. 

What  will  be  carried  forward  are  two  ideas:  1)  In  order  to 
establish  understanding,  it  will  be  important  to  provide  a 
framework  or  model,  provide  explanations  within  that 
framework,  and  connect  that  framework  to  other  more  familiar 
frameworks  and  models,  and  2)  Multiple  levels  of  understanding 
are  possible  and  depth  of  understanding  might  be  measured  in 
terms  of  internal  justifications  and  connections  as  well  as 
external  justifications  and  connections  (e.g.,  meaningful 
analogies  and  metaphors) . 

Instructional  Theories 

There  are  various  instructional  theories  which  contain  models 
prescribing  how  courses  should  be  designed,  including  the 
Gagne/Briggs  models  of  instruction,  Landa's  algo-heuristic 
theory  of  instruction,  Collins-Stevens '  cognitive  theory  of 
instruction,  Merrill's  Component  Display  Theory,  Reigeluth's 
Elaboration  Theory,  functional  decomposition  and  functional 
context  theories,  and  scenario-based  models  (Reigeluth,  1983) . 

What  are  most  noticeably  lacking  are  prescriptions  to  follow 
to  determine  when  a  particular  model  would  be  most  useful  and 
exactly  what  is  entailed  in  a  particular  application.  Merrill 
has  provided  a  few  high  level  prescriptions  for  an  authoring 
system,  but  they  are  too  general  to  be  directly  helpful 
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(Merrill,  1987) .  Reigeluth  claims  to  solve  this  problem  for 
three  knowledge  types  (conceptual,  procedural,  and 
theoretical)  ,  but  his  conclusions  are  untested  and  do  not 
provide  clear  directions  to  select  an  organizing  strategy 
(Reigeluth,  1983)  . 

The  Integrated  AIDA  Approach 

The  strategy  proposed  for  AIDA  is  to  admit  that  much  useful 
work  has  been  done  in  theory  of  instructional  design,  select 
the  best  models  and  guides  for  their  use,  incorporate  those 
models  and  guidelines  into  course  design  mini-advisors,  and 
adopt  a  selection  strategy  that  recognizes  that  a  single 
course  will  likely  involve  multiple  types  of  knowledge  and 
mastery  levels.  What  is  needed  is  an  organizing  strategy  that 
works  at  the  level  of  a  group  of  lessons,  a  course  module. 
The  size  of  the  subset  may  range  from  a  single  lesson  to  an 
entire  course,  but  the  typical  size  is  expected  to  include 
several  (3  to  7)  lessons.  Selecting  an  organizing  strategy  for 
lesson  materials  based  on  an  appropriate  instructional  model 
for  a  course  module  has  the  obvious  advantage  of  allowing  the 
design  of  AIDA  to  be  built  around  the  best  work  already 
available. 


VII.  INVESTIGATIVE  ACTIVITIES 

With  regard  to  the  two  problems  in  the  area  of  determining  the 
state  of  instructional  design  science,  there  were  four 
research  activities:  1)  a  review  of  current  and  classical 
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attempts  to  integrate  theories  of  knowledge,  learning,  and 
instruction,  2)  participation  in  the  1989  ITS  Conference 
hosted  jointly  in  San  Antonio  by  AFHRL/IDI  and  Southwest 
Research  Institute,  3)  participation  in  the  ITS  Workshop  held 
as  part  of  the  Software  Engineering  Institute's  1989  Summer 
Conference  at  Carnegie-Mellon  University,  and  4)  participation 
in  the  Utah  State  University  Annual  Summer  Instructional 
Technology  Institute  on  "Using  Artificial  Intelligence  in 
Education:  Computer  Based  Tools  for  Instructional  Design." 

With  regard  to  the  second  set  of  problems  in  the  area  of 
determining  the  role  of  artificial  intelligence  in  AIDA,  there 
were  two  research  activities:  1)  a  review  of  current 
literature  in  the  area  of  artificial  intelligence  and 
instructional  design,  and  2)  a  presentation  of  "The 
Theoretical  Limitations  of  Neurocomputing"  at  the  1989 
Jacksonville  State  University  Neural  Networks  Lecture  Series. 

With  regard  to  the  third  set  of  problems  in  support  of  the 
Laboratory's  AIDA  Task  Order  Contract  Number  F33615-88-C-0003 , 
there  were  three  investigative  activities:  1)  pre-planning 
and  coordination  with  AFHRL/IDC  in  drafting  documents 
pertinent  to  the  meetings,  2)  attendance  and  participation  in 
the  meetings,  and  3)  presenting  an  analysis  of  the  meetings  to 
AFHRL/IDC. 


VIII.  FINDINGS 


The  first  set  of  research  activities  were  an  attempt  to 


determine  the  state  of  instructional  design  science. 
Basically  the  activities  involved  a  review  of  the  relevant 
literature  and  participation  in  a  number  of  workshops  and 
seminars  about  state-of-the-art  instructional  design  systems. 

The  Classical  View 


In  reviewing  the  literature  an  attempt  was  made  to  identify 
explicit  attempts  to  integrate  theories  of  knowledge, 
learning,  and  instruction.  The  only  clearly  integrated  and 
synthesized  theory  was  found  in  the  classical  literature  of 
Plato's  dialogues,  especially  the  Meno  and  the  Theatetus 
(Jowett,  1871)  , 

Plato's  theory  is  that  learning  is  a  process  of  remembering 
what  the  soul  has  learned  in  previous  incarnations. 
Instruction  then  becomes  a  two  step  process.  First,  cause  the 
learner  to  become  aware  that  he  has  forgotten  something 
important,  such  as  the  meaning  of  virtue.  Then,  by  a  series 
of  analogies  and  reminders,  elicit  from  the  learner  what  he 
has  forgotten.  One  can  see  a  certain  similarity  between  these 
steps  as  practiced  by  Socrates  and  Robert  Gagne's  Nine  Events 
of  Instruction,  which  include  gaining  the  learner's  attention, 
informing  him  of  the  objective,  stimulating  recall,  etc. 
(Gagne,  1985) . 

Plato's  theory  of  knowledge  involves  two  key  aspects:  1) 
knowledge  involves  certainty,  and  2)  behavior  is  a  measure  of 
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knowledge.  Making  the  correct  claims  about  virtue  is  not 
sufficient  to  establish  that  one  knows  what  virtue  is.  One 
must  also  behave  in  such  a  way  that  it  is  obvious  that  one 
understands  the  meaning  of  virtue,  and,  of  course,  Socrates 
had  very  high  standards.  What  is  most  interesting  is  to  see 
the  behavioral  component  in  this  early  view. 

A  Modern  Holistic  Account 

Most  modern  theorists  will  reject  certain  aspects  of  this 
integrated  theory  (e.g.,  immortal  souls  or  infinite 
reincarnations) .  However,  the  integrated  view  is  important. 
In  the  course  of  elaborating  advanced  learning  theories,  we 
will  need  to  take  into  account  a  complete  account  of  a  person. 
It  will  not  be  sufficient  to  examine  just  the  brain  or  just 
behavior  or  just  verbal  responses.  One  modern  version  of  such 
an  attempt  to  provide  a  holistic  and  integrated  account  of 
learning  can  be  found  in  Michael  Arbib's  In  Search  of  the 
Person;  Philosophical  Explorations  in  Cognitive  Science 
(Arbib,  1985) . 


Limits  of  Current  Systems 

In  examining  current  instructional  design  systems  and  the 
development  of  CBI,  several  noteworthy  items  were  discovered. 
First,  the  history  of  CBI  reveals  a  bias  towards  the  DDD  phase 
of  instructional  design.  Little  emphasis  has  been  given  to 
FEA  and  REA.  Within  the  DDD  phase,  the  historical  trend  has 
been  to  first  emphasize  development  and  delivery  and,  almost 
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as  an  afterthought,  to  emphasize  design.  Nearly  all 
assistance  in  courseware  development  is  now  in  the  form  of 
unintelligent  but  highly  sophisticated  authoring  tools  that 
require  specialized  knowledge  to  use  and  assume  a  mastery  of 
the  principles  of  instructional  design. 

Moreover,  advances  in  technology  have  made  possible  the 
incorporation  of  sophisticated  graphics,  video,  and  sound. 
These  added  complexities  have  aggravated  the  ease-of-use 
problems  with  CBI  development  systems.  The  design  phase  has 
yet  to  receive  adequate  attention  in  terms  of  automated  tools 
and  techniques  (Montague  et  al.,  1983).  There  are  mouse- 
driven  graphics  editors  and  the  like,  but  there  are  very  few 
aids  designed  specifically  to  advise  authors  how  best  to 
organize  and  deliver  course  materials  in  a  computer-based 
environment. 


The  Need  for  AIDA 

Roughly  stated,  what  needs  to  occur  with  regard  to  courseware 
authoring  systems  is  to  provide  additional  tools  to  support 
the  design  phase  of  ODD  as  well  as  development,  delivery,  and 
the  other  two  phases  (FEA  and  REA) ,  and  to  integrate  all  these 
tools  in  an  easy  to  use,  intuitive  system  (see  Appendix) . 

Integrating  Theories 

In  response  to  the  questions  involving  developing  an 
integrated  instructional  theory  (Can  theories  of  learning. 
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knowledge,  and  instruction  be  integrated?,  and  What  theories 
are  being  used?) ,  it  is  possible  to  provide  an  integrated 
account  of  learning,  instruction  and  knowledge  (Plato  has  done 
so)  ,  but  modern  theorists  typically  do  not  make  explicit  the 
connections  between  and  among  knowing,  learning,  and 
instructing.  Examples  of  current  instructional  theories 
include  Gagne's  Nine  Events  of  Instruction,  Merrill's 
Component  Display  Theory,  and  Reigeluth's  Elaboration  Theory. 

The  Role  of  AI 

The  second  set  of  research  activities  involved  an  exploration 
of  the  role  for  artificial  intelligence  in  instructional 
design.  This  exploration  followed  two  paths:  l)  neural 
networks,  and  2)  expert  systems,  including  intelligent  tutors. 
These  are  the  two  major  areas  of  artificial  intelligence 
applications,  so  this  was  a  natural  course  to  pursue. 

Neural  Networks 

While  there  is  a  great  deal  of  current  activity  in  the  area  of 
neural  networks,  there  does  not  appear  to  be  any  direct  or 
immediate  application  in  the  area  of  instructional  design. 
Neural  nets  are  basically  learning  automata,  inspired  by  the 
workings  of  the  human  brain.  Areas  of  application  include 
pattern  recognition,  speech  recognition,  and  robotics 
(Soucek,  1988)  . 


As  automated  learning  environments  become  more  conversational 
with  improving  speech  recognition  and  generation  technologies, 


it  will  be  natural  to  reconsider  the  role  for  neural  networks, 
especially  in  the  area  of  speech  recognition.  Conversational 
systems  will  introduce  an  important  new  level  of  interactivity 
into  CBI.  As  a  consequence,  monitoring  progress  in  neural 
network  speech  recognition  technology  should  be  a  priority 
project  at  the  Laboratory. 

Expert  Systems 

The  story  with  regard  to  expert  systems  is  different,  however. 
Expert  systems  have  become  commercially  successful  within  the 
lasc  ten  years  in  a  variety  of  domains.  It  is  possible  to 
divide  expert  systems  into  two  categories:  1)  Diagnostic 
systems  that  proceed  from  symptoms  backwards  toward  a 
relatively  small  set  of  causes,  and  2)  Planning  systems  that 
proceed  forward  from  a  given  set  of  restrictions  to  a 
projected  set  of  outcomes.  In  both  types  of  systems  there  is 
a  similar  overall  architecture  involving  a  set  of  rules,  a  set 
of  facts,  and  an  inference  mechanism  (Luger,  1989) . 

Early  expert  systems  were  written  in  LISP  or  PROLOG  or  another 
high  level  computer  language.  The  challenge  for  computer 
science  was  to  design  effective  inference  mechanisms.  As 
algorithms  for  the  match  and  apply  phase  of  the  inference 
engines  became  more  complex,  higher  level  tools  were 
developed.  For  example,  the  RETE  pattern  matching  algorithm 
developed  by  Charles  Forgy  at  Carnegie  Mellon  University  to 
match  facts  with  the  IF-components  of  rules  was  incorporated 
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in  the  OPS  series  of  production  system  development  tools 
(Giarratano,  1989) . 

There  now  exist  a  number  of  expert  system  shell  development 
languages  which  hide  the  complexities  of  the  inference 
mechanisms  from  the  user.  The  idea  behind  these  tools  is  to 
make  the  technology  of  expert  systems  available  to  domain 
experts  and  require  minimal  knowledge  of  artificial 
intelligence  and  programming.  Examples  of  such  systems 
include  ART,  EXSYS,  GURU,  KEE,  and  Personal  Consultant 
(Firebaugh,  1988) . 

Expert  Systems  in  Courseware  Development 

What  should  occur  with  courseware  authoring  systems  is  exactly 
what  has  been  happening  with  regard  to  expert  system 
development  tools.  That  is  to  say,  easy  to  use  tools  for 
domain  experts  need  to  be  developed. 

In  addition,  incorporating  expert  systems  into  the 
instructional  design  process  appears  both  possible  and 
worthwhile,  especially  in  a  typical  military  setting  where  the 
instructional  designer  has  had  little  formal  training  in  the 
instructional  design  process.  Diagnostic  expert  systems  have 
been  successfully  incorporated  in  the  form  of  intelligent 
tutors,  which  analyze  student  responses  and  adjust  courseware 
delivery  in  accordance  with  a  diagnosis  of  a  student's 
(mis) understanding  based  on  comparison  with  a  model  of  an 
expert's  understanding  (see  Section  II) .  Intelligent  tutoring 
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systems  have  been  shown  to  be  effective  in  specific  and 
restricted  domains,  but  they  are  expensive  to  dc’^elop  and  to 
implement  (Wenger,  1987) . 

Intelligent  Instructional  Design  Guidance 

Another  challenge  is  to  develop  expert  system  planners  to  aid 
courseware  designers  to  develop  consistently  effective  course 
materials  in  a  cost-effective  and  systematic  manner  for  a 
variety  of  subject  matter  domains  and  knowledge  types  (e.g., 
declarative,  procedural,  and  causal).  There  are  a  few 
intelligent  systems  to  aid  course  authors  design  course 
materials  for  declarative  or  factual  knowledge  types. 

For  example,  the  Air  University's  Air  Command  and  Staff 
College  at  Maxwell  AFB  has  developed  a  system  called  SOCRATES, 
which  is  intended  to  assist  subject  matter  experts  in 
organizing  course  material  into  effective  lesson  materials. 
SOCRATES  is  based  on  Robert  Gagne's  Nine  Events  of  Instruction 
and  David  Merrill's  Component  Display  Theory  (Doucet,  1988). 

As  mentioned  earlier,  ARI  partially  funded  a  system  developed 
by  David  Merrill  at  Utah  State  University  called  ID  Expert.  ID 
Expert  is  an  expert  consultation  system  for  the  design  and 
development  of  instruction. 

Kent  Gustafson  at  the  University  of  Georgia  has  developed  a 
system  for  Apple  Training  Systems  called  IDioM.  IDioM  is 
intended  to  provide  support  for  the  entire  instructional 
design  process.  IDioM  is  built  on  HyperCard  and  is  probably 
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the  friendliest  and  easiest  to  use  system  of  those  mentioned. 
It  is  also  one  of  the  very  few  which  acknowledges  the  FEA 
phase  of  instructional  design.  IDioM's  intelligence  is 
primarily  restricted  to  the  selection  of  appropriate  templates 
for  instruction  based  on  input  parameters,  but  this  is  one 
useful  form  of  machine  intelligence,  and  it  does  automate  one 
part  of  the  ISD  process.  IDioM's  user  interface  has  been 
upgraded  and  the  product  has  been  renamed  ID  Bookshelf. 

There  are  a  few  other  systems  under  development,  such  as  the 
Alberta  Research  Council's  Expert  CML,  ARI's  Automated  Systems 
Approach  to  Training  (ASAT) ,  Ford  Aerospace  Corp.'s  The 
Instructional  Prescriptions  Systems  (TIPS),  the  Navy's 
Authoring  Instructional  Materials  (AIM),  and  Xerox-PARC's  IDE. 
The  effectiveness  of  all  of  these  systems  has  yet  to  be  fully 
evaluated.  Furthermore,  no  system  exists  as  yet  to  advise 
course  authors  how  best  to  design,  develop,  and  deliver 
materials  that  involve  multiple  knowledge  types  for  a  variety 
of  subject  matter  domains  and  instructional  settings.  Part  of 
the  challenge  for  AIDA  is  to  accomplish  this  for  computer- 
based  settings  and  to  provide  a  full  range  of  automated  tools 
to  support  all  major  phases  of  the  ISD  process  in  an  easy  to 
use,  flexible,  and  intuitive  system. 

The  answers  to  the  two  questions  about  the  role  for  artificial 
intelligence  are  now  obvious:  1)  There  is  a  role  for  expert 
planning  systems  in  the  instructional  design  process,  and 
possibly  a  role  for  an  intelligent  tutor  to  teach 
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instructional  design,  and  2)  There  is  no  immediate  role  for 
neural  networks  in  the  instructional  design  process,  although 
there  may  be  a  role  in  the  area  of  speech  recognition  in  the 
not-so-distant  future. 


A  Crucial  Issue 

One  of  the  most  fundamental  issues  yet  to  be  resolved  is  how 
to  represent  (if  it  possible)  the  knowledge  of  experts, 
specifically  instructional  design  experts  (Winograd,  1986) . 
Because  expert  systems  are  intended  to  emulate  human  experts 
in  the  sense  of  producing  results  consistently  similar  to 
those  produced  by  human  experts,  it  will  be  necessary  to 
identify  human  e..perts  in  the  area  of  instructional  design  and 
to  measure  the  effectiveness  of  automated  design  tools  against 
the  effectiveness  of  those  experts. 


IX.  AIDA  DESIGN  REQUIREMENTS 

The  following  requirements  indicate  the  basic  design  concerns 
for  AIDA,  given  the  problem  context,  the  problem  areas,  and 
the  fundamental  concerns  just  discussed: 

1.  Proceed  according  to  a  standard  design  philosophy  that 
maintains  consistency  of  materials  and  presentations.  The 
issues  with  regard  to  standard  design  philosophy  include  but 
are  not  restricted  to:  standard  selection  processes,  dynamic 
default  settings,  multiple  and  dynamic  levels  of  assistance, 
the  standard  use  of  mini-advisors  with  relatively  small 
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special-purpose  rule  sets,  standard  ISD  documentation  and 
materials  for  students,  standard  design  for  editors,  a 
standard  graphical  user  interface  (GUI) ,  and  standard 
interfaces  to  other  modules  and  commercial  systems.  Because 
the  motivation  for  a  standard  design  philosophy  is  the 
integration  of  tools,  the  design  of  standard  databases  with 
information  about  existing  schools,  courses,  delivery  systems, 
and  students  is  necessary. 

2.  Proceed  within  the  framework  of  a  theory  of  knowledge. 
The  suggested  theory  here  is  JCB  because  it  is  a  practical 
modification  to  an  established  epistemological  view.  The 
modifications  are  intended  to  provide  some  connection  to  both 
learning  and  instruction  theories.  The  emphasis  on  WELL- 
justified  and  WELL-connected  beliefs  provides  a  basis  for 
guiding  understanding  as  well  as  a  basis  for  evaluating 
understanding.  The  idea  of  a  web  of  beliefs  and  a  variety  of 
knowledge  types  lends  itself  to  both  instructional  and 
learning  interpretations. 

3.  Proceed  within  the  framework  of  a  modularized  theory  of 
instruction.  The  theory  adopted  initially  is  that  Reigeluth's 
Elaboration  Theory,  Merrill's  Component  Display  Theory,  the 
Gagne/Briggs  models  of  instruction,  and  other  theories  offer 
meaningful,  useful,  and  potentially  effective  organizing 
strategies  for  course  materials.  What  is  most  needed  is  an 
algorithm  to  determine  when  to  apply  which  model.  The  reason 
the  selection  algorithm  has  not  bean  established  is  that  the 
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assumption  was  that  the  strategy  would  apply  course-wide; 
this,  at  least,  is  Reigeluth's  assumption.  It  is  exactly  the 
assumption  of  a  single  course-wide  organizing  strategy  that 
this  design  rejects.  The  proposal  here  is  that  an  appropriate 
and  effective  selection  can  be  made  for  a  course  module. 

4.  Proceed  with  a  design  using  these  functional  modules:  a) 
"macro”  module  to  assist  in  refining  course  goals  into  modules 
and  objectives,  b)  "mini"  module  to  assist  in  the  selection  of 
appropriate  strategies  for  selecting,  organizing,  and 
sequencing  course  module  materials  ,  and  c)  "micro"  module  to 
assist  in  the  selection  and  sequencing  of  materials  within  a 
lesson.  Each  module  would  eventually  have  access  to  databases 
containing  course  information,  templates,  examples,  etc.  Each 
module  would  also  have  access  to  a  set  of  mini-editors  to 
assist  in  the  creation,  alteration,  and  incorporation  of 
lesson  materials  into  a  course.  AIDA  should  be  designed  so 
that  it  could  support  a  wide  range  of  instructional  settings, 
including  the  classroom  lecture  setting,  a  computerized  and 
networked  lab  setting  with  full  support  for  CMI ,  and 
specialized  settings  with  ITSs  and  simulations.  In  short,  the 
AIDA  system  would  consist  of  a  set  of  modules  which  all 
supported  multiple  modes  of  operation,  multiple  and  dynamic 
levels  of  assistance,  user-transparent  interfaces,  and  other 
features  prescribed  by  the  standard  design  philosophy. 
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Depicted  below  is  a  conceptual  model  of  AIDA: 
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Functional  Model  of  AIDA 


AIDA  would  make  available  a  variety  of  operating  modes.  For 
example,  at  the  highest  level  there  might  be  different  views 
of  the  system  for  training  managers,  instructional  designers 
and  developers,  and  students.  For  the  developer,  there  would 
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be  inodes  for  entering  material  (text,  graphics,  etc.)  into 
lesson  templates,  for  storyboarding  and  linking  material,  and 
for  testing  the  delivery  of  the  lessons  created. 

Depicted  below  is  a  conception  of  an  AIDA  menu  that  would  be 
consistent  with  the  previous  recommendations: 
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Figure  2.  An  AIDA  Menu 
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X.  RECOMMENDATIONS 


The  primary  conclusion  of  this  research  effort  is  that  it  is 
worth  pursuing  the  design  of  AIDA.  Significant  issues  need  to 
be  resolved,  including  the  desirability,  feasibility,  and 
costs  of  an  integrated  system  of  modules  sharing  access  to 
common  databases,  common  selection  processes,  files,  mini¬ 
editors  and  mini-advisors.  If  one  begins  with  the  possibility 
of  such  a  system,  then  one  is  led  to  the  need  for  a  standard 
design  philosophy  to  guide  related  efforts. 

AFHRL/IDC  should  consider  the  following  specific  actions; 

1.  Assign  a  team  with  expertise  in  theory  of  knowledge, 
cognitive  and  behavioral  theories  of  learning,  AI,  integrated 
computer  systems,  and  curriculum  and  instructional  design  with 
the  task  of  developing  the  specifications  for  a  standard 
design  philosophy  for  all  tools  developed  to  automate  and 
assist  the  instructional  process. 

2.  Assign  a  team  with  similar  expertise  with  the  task  of 
refining  the  design  requirements  for  AIDA.  The  design 
requirements  need  to  be  refined  to  reflect  such  things  as  the 
selection  algorithm  to  be  used  to  organize  material  at  the 
course  module  level,  the  specific  structure  and  modules 
required  for  AIDA,  a  minimal  subset  of  modules  to  be 
implemented  in  a  prototype,  and  initial  rule  sets  to  be  used 
in  the  mini-advisors. 
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3.  Have  those  documents  (standard  design  philosophy  and  AIDA 
design  requirements)  critically  reviewed  by  a  team  of 
specialists  from  leading  academic  research  centers  and  from 
key  military  training  centers. 

4.  Have  the  reviewers  meet  in  a  forum  and  share,  clarify,  and 
refine  their  critiques  of  the  two  documents.  Detailed  reviews 
of  SOCRATES  and  similar  systems  should  be  provided  so  as  to 
gain  the  most  recent  and  pertinent  lessons  learned. 

5.  Produce  the  final  standard  design  philosophy 
specifications  and  design  requirements  for  AIDA. 

6.  Initiate  work  on  related  projects  rising  the  standard  design 
philosophy  as  a  guiding  requirements  standard. 

7.  Design  and  develop  a  prototype  AIDA  system  with  a  minimal 
subset  of  functions  and  models,  and  test  it  in  controlled 
settings. 

8.  Evaluate  progress  on  AIDA  and  related  projects. 

It  should  be  noted  that  the  Laboratory  has  undertaken  all  of 
the  above  tasks  with  the  exception  of  those  involving  the 
development  of  a  standard  design  philosophy. 

The  Laboratory  should  definitely  continue  with  the  design  and 
development  of  a  prototype  AIDA  system.  It  is  now  time  to 
shift  the  emphasis  back  to  the  less  theoretical  and  more 
practical  design  and  development  issues.  The  reed  for  a 
guiding  and  coherent  theory  is  obvious,  but  the  design  and 


development  of  AIDA  need  not  await  the  validation  of  such  a 
theory.  Indeed,  AIDA  is  intended  partly  as  a  tool  to  use  in 
refining  and  integrating  theories  of  knowledge,  learning,  and 
instruction. 

The  Laboratory  should  not  abandon  attempts  to  make  AIDA  an 
intelligent  system.  There  appears  to  be  no  immediate  role  for 
neural  networks  in  AIDA.  However,  there  are  potential  roles 
for  a  number  of  expert  system  planners  throughout  the  ISD 
process.  Of  special  concern  is  the  potential  role  for  expert 
system  technology  to  facilitate  the  DDD  phase  of  courseware 
development. 

With  regard  to  prototyping,  serious  consideration  should  be 
given  to  developing  the  prototype  using  an  object-oriented 
design/hyper-media  development  system  such  as  HyperCard  on  the 
Apple  Macintosh,  NextStep  on  the  NeXT  computer  system,  or 
Hewlett-Packard's  NewWave  with  Windows  3.0  on  an  80286/80386 
system.  The  prototype  system  need  not  use  the  same  hardware 
or  software  as  the  final  target  system.  The  critical  choices 
for  a  prototype  development  system  are  cost  and  ease  of 
development  effort.  The  point  of  the  prototype  is  to 
illustrate  the  functionality  of  the  proposed  system  in  an 
effort  to  prove  the  concept  is  workable.  It  should  be  noted 
that  adoption  of  this  prototyping  recommendation  involves 
abandoning  the  Instructional  Support  System  (ISS)  as  a  test¬ 
bed  delivery  system  for  the  prototype. 
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APPENDIX 


NEEDS  ASSESSMENT  REPORT 

Visit  to  Lowry  AFB  Technical  Training  Center  (3400  TCHTW) 

2-6  Oct  1989 

Drs.  Dan  Muraida  &  Mike  Spector  (AFHRL/IDC) 

I .  Purpose 

The  purpose  of  the  visit  was  to  conduct  a  preliminary  needs 
assessment  for  the  Advanced  Instructional  Design  Advisor 
(AIDA) .  The  motivating  objective  was  to  involve  experienced 
Air  Force  instructional  designers,  developers,  training 
managers,  instructors,  and  other  relevant  personnel  in  the 
early  design  work  on  AIDA  to  insure  that  AFHRL/IDC  research 
and  development  efforts  are  directed  toward  practical  and 
useful  goals. 

II.  Agenda 

2  Oct  89 

1500  -  1700  3400  TCHTW/TTO  (Wing  Tech  Tng) 

3  Oct  89 

0900  -  1100  3450  TCHTG/TTMYF  (Tng  Dev  Branch) 

3450  TCHTG/TTMYF  (Instructors) 

1300  -  1500  3400  TCHTW  (Training  Managers) 

4  Oct  89 

0900  -  1100  3450  TCHTG/TTMYA  (ATLAS  F-16) 

1300  -  1500  3460  TCHTG/TTMTN  (BMIS) 

5  Oct  89 

0900  -  1100  3430  TCHTG/TTMA  (Space  Systems) 

1300  -  1500  3400  TCHTW/TTF  (CBT  Tng  &  Fac  Dev) 

6  Oct  89 

0900  -  1100  3440  TCHTG/TTMXS-Z  (MS&D  Exp  Tng) 
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III.  Format 


The  trip  was  coordinated  through  Lt.  Col.  Hector  Acosta  (ATC 
HQ/XPC) .  All  meetings  were  arranged  by  Brian  Dallman  (3400 
TCHTW/TTOZ) .  AFHRL/IDC  representatives  were  Drs.  Dan  Muraida 
and  Mike  Spector.  Capt.  Jim  coward  (ATC  HQ/XPCRI)  represented 
ATC  headquarters. 

Brian  Dallman  arranged  meetings  with  small  groups  of  key 
personnel  involved  in  all  phases  of  instructional  design, 
development,  and  support.  Groups  varied  in  size  from  4  to  14. 
Discussion  was  guided  by  a  needs  assessment  guide.  A  survey 
was  not  formally  administered  because  it  was  felt  that  more 
meaningful  information  and  insights  could  be  gathered  through 
informal  discussions  facilitated  by  Brian  Dallman. 


IV.  Summaries 

A.  2  oct  89  1500  -  1700  3400  TCHTW/TTO  (Wing  Tech  Tng) 

Participants  at  this  ireeting  were  Brian  Dallman,  the  Training 
Wing's  Technical  Applications  Officer,  Fred  Norman,  Chief  of 
the  Training  Wing  Operations  Division  Plans  Branch,  Capt. 
Coward,  Dan  Muraida,  and  Mike  Spector.  Main  topics  of 
discussion  included  the  general  process  of  designing, 
developing,  and  delivering  courses  in  the  training  wing,  the 
pertinent  ATC  regulations  and  documentation,  the  organization 
of  the  training  wing,  the  interrelationships  of  the  various 
groups  and  branches,  and  the  relevant  decision  making 
processes. 

Documentation  pertinent  to  course  development  includes  1) 
training  plans,  which  contain  specialty  training  standards 
(STSs) ,  and  course  charts  indicating  the  general  sequence  of 
instruction,  and  2)  plans  of  instruction  (POIs) ,  which  contain 
measurements,  curriculum  descriptions,  study  guides,  and 
lesson  plans.  The  Plans  Branch  puts  controls  in  place  and 
oversees  compliance  with  ATC  regulations  through  various 
training  managers.  The  training  managers  interact  with  the 
various  Training  Development  Branches  (TDBs)  in  the  Training 
Groups.  The  TDBs  are  responsible  for  the  development  and 
management  of  specific  training  materials  and  instruction. 
Training  specialists  and  instructors  assigned  to  various 
projects  are  responsible  for  designing,  developing, 
delivering,  and  modifying  course  materials.  Organization  of 
the  training  wing  is  in  accordance  with  AFR  23-40. 

The  decision  to  train  is  typically  made  on  the  basis  of 
occupational  survey  reports  (OSRs)  and  then  passed  to  the 
training  wing,  sometimes  with  constraints  (time  in  school, 
budgets,  whether  or  not  to  export  the  training,  etc.).  The 
Training  Groups  are  responsible  for  developing  course  resource 
estimates. 


The  decision  to  use  self-paced  instruction  (programioed  texts 
or  computer-based  training)  is  typically  made  in  developing 
the  training  plan  after  the  course  resource  estimate  is 
developed.  The  training  plan  is  developed  following  the  guide 
of  the  STS's  task  knowledge,  skill,  and  performance  levels. 
Plans  of  instruction  are  then  developed  to  elaborate  training 
plans.  Lesson  plans  are  then  implemented  and  the  course 
taught.  Evaluations  typically  occur  in  the  field  about  6 
months  after  graduation  from  the  school. 

Most  of  the  training  is  developed  to  be  delivered  in  a  group- 
paced,  stand-up  setting  with  small  classes  (8  students) .  A 
few  interactive,  self-paced  courses  are  being  developed. 
Typical  hours  of  development  per  course  hour  for  self-paced 
instruction  are  600  to  700. 


B.  3  Oct  89  0900  -  1100  3450  TCHTG/TTMYF  (Tng  Dev  Branch) 

3450  TCHTG/TTMYF  (Instructors) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 
Coward,  Dr.  Muraida,  Dr.  Spector,  3  training  specialists  from 
a  TDB,  and  3  instructors.  These  training  specialists 
represented  the  Avionics  Training  Division  which  had  recently 
put  10  new  courses  in  place. 

Two  of  the  TDB  personnel  had  come  straight  from  an  overseas 
flight-line  assignment  to  the  TDB  with  no  prior  specific 
schooling  or  course  authoring  experience.  One  instructor 
admitted  that  he  was  not  a  qualified  subject  matter  expert 
(SME)  due  to  equipment  upgrades.  TDB  personnel  indicated  that 
strict  adherence  to  ATC  Instructional  Systems  Design  (ISD) 
procedures  caused  some  problems  and  misunderstanding. 
Specifically,  the  ISD  sequence  from  objectives  to  tests  to 
materials  was  not  generally  accepted  or  understood. 
Instructors  reported  that  standards  were  lax  (e.g.,  80%  on  an 
open  book  exam)  and  the  wash-out  rate  was  non-existent. 
Training  quality  reports  from  the  field  were  not  timely,  were 
not  detailed,  and  lacked  constructive  criticism,  according  to 
TDB  personnel. 

Both  TDB  personnel  and  instructors  reported  that  instructional 
design  decisions  were  made  above  the  training  group  level 
(training  managers  at  the  wing,  at  ATC  headquarters,  or  at  the 
using  commands) .  TDB  personnel  indicated  that  lesson 
sequencing  was  done  almost  totally  in  accordance  with  the 
sequence  reflected  in  the  STS. 


C.  3  Oct  89  1300  -  1500  3400  TCHTW  (Training  Managers) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 
Coward,  Dr.  Muraida,  Dr.  Spector,  and  6  training  managers 
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(TMs) ,  all  but  one  of  whom  were  civilians,  whereas  most  of  the 
training  specialists  in  the  training  groups  were  NCOs.  The 
primary  focus  of  this  meeting  was  decision-making  processes 
affecting  course  design  and  development. 

The  training  managers  reported  that  the  TDBs  do  not  develop 
instruction.  Instruction  is  developed  by  SMEs  who  are  not 
instructional  designers.  TMs  may  have  instructional  design 
expertise  but  do  not  dictate  instructional  design.  This  group 
of  TMs  reported  that  important  instructional  design  and 
development  decisions  were  made  at  systems  acquisition  time  or 
by  System  Program  Offices  (SPOs) ,  especially  for  equipment- 
based  courses.  TMs  did  not  interact  directly  with  the  SMEs; 
rather  they  interacted  with  the  TDBs.  TMs  described  ISD 
documentation  requirements  as  a  hassle  and  indicated  that  they 
had  little  authority  to  make  instructional  design  or 
development  decisions. 


D.  4  Oct  89  0900  -  1100  3450  TCHTG/TTMYA  (ATLAS/F-16) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 
Coward,  Dr.  Muraida,  Dr.  Spector,  and  6  training  specialists 
assigned  to  the  ATLAS/F-16  &  F-111  IVD  projects.  The  focus  of 
this  discussion  was  on  instructional  design  and  lesson 
development  in  an  IVD  setting. 

The  3306  TES  set  the  training  requirements,  including 
decisions  about  training  resources.  These  decisions  were 
passed  along  to  TMs  who  passed  them  to  the  TDBs.  TDB 
personnel  reported  being  left  out  of  the  decision-making 
process.  Some  SMEs  did  not  agree  with  the  STS  requirements  as 
there  were  minimal  requirements  for  hands-on  knowledge;  based 
on  their  own  field  experience  they  believed  that  hands-on 
knowledge  was  highly  desirable.  Course  sequencing  typically 
followed  the  outline  of  the  STS. 

Instructional  designers  reported  a  long  learning  curve  to 
master  QUEST  and  other  aspects  of  IVD  authoring.  They  used  an 
AIS  IVD  model  developed  by  AFHRL;  they  reported  that  having 
additional  IVD  lesson  models  would  have  been  helpful.  There 
was  concern  that  the  expertise  of  these  military  training 
specialists  would  be  lost  when  they  transferred  to  another 
assignment. 

Designers  indicated  that  a  guide  on  the  use  of  IVD  published 
at  Hill  AFB  and  ATC  Manual  50-4  on  CAI  were  useful.  Many 
design  decisions,  however,  were  made  by  trial  and  error.  The 
first  3  or  4  course  offerings  were  used  for  internal 
validation. 

Essential  requirements  for  the  authoring  system  in  this 
environment  were  reported  to  be  flexibility  and  complexity  in 
order  to  accommodate  the  varying  lesson  requirements.  In 


42 


addition,  due  to  frequent  modifications  to  the  aircraft, 
course  materials  were  in  a  constant  state  of  revision,  making 
ease-of-modif ication  an  important  characteristic  of  an 
authoring  system. 

The  group  reported  that  the  faculty  development  1  month  course 
on  the  QUEST  language  was  inadequate  because  it  did  not 
incorporate  any  materials  on  the  use  of  IVD.  Dr.  Ruth 
Gordon's  1  week  course  on  CAI  for  training  managers  was  judged 
outstanding,  as  was  the  AIS  training  offered  off  base. 

Designers  reported  a  general  lack  of  materials  to  guide  CBT 
development  and  that  existing  ISD  materials  were  not  relevant 
to  CBT.  They  also  reported  that  time  constraints  to  get 
courses  on-line  caused  violations  in  regular,  established 
review  procedures.  Several  time-consuming  tasks  were 
identified,  including  reviews  of  training  plans  and  course 
charts,  required  documentation  (especially  ATC  Form  10) ,  and 
formatting  disks  and  grading  tests. 


E.  4  Oct  89  1300  -  1500  3460  TCHTG/TTMTN  (BMIS) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 
Coward,  Dr.  Muraida,  Dr.  Spector,  and  4  training  specialists 
assigned  to  the  BMIS  project.  The  focus  of  this  discussion 
was  the  process  of  instructional  development  in  a  CBT 
environment. 

This  group  indicated  that  the  decision  to  use  CBT  had  been 
made  by  outsiders.  They  were  handed  the  requirement  of 
developing  exportable  courses  to  be  maintained  at  Lowry  AFB. 
They  indicated  that  the  only  specialized  military  training 
received  relevant  to  their  jobs  were  the  faculty  development 
courses  on  QUEST  (1  month)  and  CAI  (1  week)  and  these  courses 
were  superficial.  They  all  indicated  a  willingness  and 
eagerness  to  take  additional  courses  and  learn  more. 

A  great  deal  of  time  was  spent  in  flowcharting  lessons.  One 
training  specialist  reported  that  300  hours  were  spent  in 
developing  a  flowchart  for  a  45  minute  lesson  (this  includes 
the  review  process) .  Typical  development  hours  per  hour  of 
instruction  were  in  the  500  to  600  range.  Most  designers  were 
self-taught  and  learned  by  reading,  seeing  good  examples,  and 
trial  and  error. 

Several  personnel  indicated  that  decision  aids  for  media 
selection  would  have  been  useful.  Rigid  interpretation  of  ISD 
documentation  was  reported  as  a  problem  area.  Teamwork  was 
identified  as  an  essential  ingredient  of  successful  CBT 
development,  with  the  team  composed  of  an  SME,  an 
instructional  design  specialist,  and  a  programmer.  This  group 
felt  fortunate  to  have  collected  this  expertise  and  worried 
what  would  happen  when  military  personnel  transferred  out. 
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F.  5  Oct  89  0900  -  1100  3430  TCHTG/TTMA  (Space  Systems) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 
Coward,  Dr.  Muraida,  Dr.  Spector,  and  8  training  specialists 
in  the  CBT  Development  Branch.  This  group  included  several 
officers,  NCOs,  and  civilian  personnel.  They  were  operating 
in  a  new  building  with  new  computer  laboratories.  They  had 
recently  put  3  new  courses  in  place.  The  discussion  focused 
on  their  experiences  with  CBT  and  IVD. 

An  immediate  concern  about  sustaining  financial  support  for 
this  very  well-equipped  facility  was  expressed.  Although  this 
facility  was  well-equipped,  there  was  no  local  capability  for 
IVD  editing  or  simulation  editing.  In  addition,  a  local  area 
network  (LAN)  for  computer-managed  instruction  (CMI)  and 
graphics  development  software  were  cited  as  other  resources  in 
immediate  demand. 

There  was  a  great  deal  of  programming  experience  in  this 
group,  they  cited  the  need  for  additional  training  in  the  area 
of  instructional  design.  They  had  learned  CBT  design 
techniques  by  trial  and  error  and  by  examining  the  few 
available  models.  In  designing  instruction,  they  were  guided 
by  3  general  goals:  1)  involve  the  learner,  2)  make  the 
system  responsive  to  the  learner,  and  3)  make  the  material 
interest-sustaining.  There  was  no  specific  guides  for  the 
elaboration  of  any  of  these  goals. 

When  asked  about  useful  tools  in  designing  and  developing 
instruction,  this  group  cited  the  usefulness  of  IVD  editors, 
simulation  editors,  graphics  editors,  and  tools  to  automate 
the  ISD  documentation  requirements.  A  rapid  lesson  prototyper 
would  be  extremely  useful.  They  believed  guides,  manuals,  and 
models  for  CBT  were  quickly  outdated,  based  on  past 
experience. 

Story-boarding  was  cited  as  an  especially  time  consuming  task. 
Lack  of  a  central  repository  of  information  and  expertise  was 
cited  as  an  additional  problem  area.  The  importance  of 
maintaining  an  important  and  meaningful  role  for  the 
instructor  in  a  CBT  setting  was  emphasized,  although  no 
specific  or  systematic  ways  of  doing  this  were  cited. 

This  group  believed  uninformed  outsiders  were  establishing 
requirements  for  instructional  design  and  imposing  impossible 
time  constraints.  They  cited  the  lack  of  standardization  of 
such  lesson-level  things  as  screen  design  as  another  problem 
area. 


G.  5  Oct  89  1300  -  1500  3400  TCHTW/TTF  (CBT  Tng  &  Fac  Dev) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 


44 


Coward,  Dr.  Muraida,  Dr.  Spector,  and  four  representatives 
from  the  faculty  development  division  (1  officer,  1  civilian, 
and  2  NCOS) .  The  focus  of  this  meeting  was  the  support  given 
by  faculty  development  to  the  various  training  groups. 

Faculty  development  personnel  indicated  that  there  was  minimal 
support  for  training  in  the  area  of  CBT  because  there  needed 
to  be  a  documented  30%  usage  level  before  regular  and  standard 
support  could  be  provided.  Current  courses  aimed  at  in-coming 
SMEs  who  end  up  doing  most  of  the  instructional  design 
emphasized  the  required  ATC  ISD  documentation.  This  group 
believed  it  was  an  invalid  assumption  and  a  mistake  to  think 
that  any  of  the  faculty  development  courses  would  turn  an  SME 
into  an  instructional  designer.  They  believed  that  TDB 
personnel  should  be  responsible  for  most  of  the  instructional 
design  and  let  the  SMEs  focus  on  actual  instruction  and 
maintaining  their  subject  matter  expertise. 

The  faculty  development  group  said  that  they,  too,  needed  more 
training  in  the  area  of  instructional  design.  A  central 
repository  of  bad  examples  of  instructional  design,  as  well  as 
a  collection  of  good  examples,  was  cited  as  a  desirable  tool. 
They  believed  that  the  training  groups  needed  more  guidance  in 
the  area  of  instructional  design  and  suggested  established 
design  teams  and  consolidating  existing  expertise  and  lessons 
learned  in  some  manner  (expert  support  team  or  an  expert 
system) . 


H.  6  Oct  89  0900  -  1100  3440  TCHTG/TTMXS-Z  (MS&D  Exp  Tng) 

Participants  in  this  meeting  included  Brian  Dallman,  Capt. 
Coward,  Dr.  Muraida,  Dr.  Spector,  and  14  members  of  the  3440th 
Training  Group  who  were  involved  in  developing  exportable  CBT 
training.  The  focus  of  discussion  in  this  meeting  was  their 
experiences  in  adapting  ISD  to  the  exportable  CBT  arena. 

This  group  of  training  specialists  had  worked  closely  with  the 
using  command  to  establish  several  specific  principles  of 
instructional  design  appropriate  to  their  situation.  For 
example,  they  required  a  student  interaction  every  3  to  5 
frames.  In  determining  appropriate  feedback,  they  asked  3 
questions,  categorized  as  easy,  medium,  and  hard,  and  then 
branched  based  on  responses.  They  attempted  to  place  all 
fundamentals  and  basics  in  the  first  block  of  instruction. 
The  group  was  happy  to  have  this  kind  of  specific  guidance  and 
did  not  question  its  universal  application  in  their  setting. 

The  ISD  model  that  had  been  adapted  to  their  setting  involved 
these  steps:  1)  task  breakdown  into  elements,  2)  tiger  team 
review  of  the  task  breakdown,  3)  identifying  instructional 
outcomes  for  each  task  and  appropriate  methodologies  (e.g., 
remediation,  continuance,  etc.),  4)  developing  tutorial 
materials  with  interactions  every  3  to  5  frames,  5)  practice 


45 


with  the  particular  task  components,  and  6)  mastery  testing  of 
the  whole  task. 

The  group  expressed  dissatisfaction  with  fitting  the  whole- 
task  mastery  model  into  rigidly  interpreted  ATC  school-house 
models  of  satisfactory  performance  (e.g.,  70%  might  be  a  pass 
in  a  typical  stand-up  setting).  In  general,  the  group 
believed  that  many  of  the  requirements  being  imposed  by  wing 
training  personnel  were  not  appropriate  in  their  setting. 
They  also  believed  that  they  had  not  received  the  appropriate 
training  for  their  task  and  had  been  thrown  into  an  unfamiliar 
and  demanding  setting. 

In  spite  of  these  perceived  obstacles,  this  group  expressed  a 
can-do  attitude  and  a  principled,  systematic  approach  to  CBT 
design.  They  developed  elaborate  task  hierarchies  and 
reviewed  these  hierarchies  in  a  group  setting.  In  developing 
the  hierarchies  they  were  guided  by  this  question:  If  you 
were  a  job  site  supervisor,  what  would  you  like  this  person  to 
know  at  the  end  of  this  training?  They  made  minimal 
assumptions  about  student  knowledge.  They  grouped  material 
into  40  minute  lessons. 

The  most  time-consumirfg  task  was  the  review  process  for  lesson 
materials.  Reviewers  were  viewed  as  inconsistent  and  the 
linear  review  process  caused  unnecessary  delays.  Consistent 
standards  for  lesson  materials  had  not  been  carefully 
elaborated.  Story-boarding  and  flowcharting  was  done  by  hand 
and  was  also  time-consuming.  Automated  ISD  documentation  was 
definitely  desired,  especially  in  a  manner  that  put  the  ISD 
process  in  the  background  and  allowed  the  training  specialists 
to  focus  on  the  design  and  development  of  lesson  materials. 

In  addition,  they  believed  that  they  were  operating  with  a 
standard  of  270  hours  of  development  per  hour  of  instruction 
imposed  by  the  training  wing.  Wing  personnel  present 
indicated  that  this  had  been  an  unfortunate  misunderstanding. 
Everyone  believed  that  270:1  was  a  totally  unrealistic  ratio, 
especially  for  novices  working  in  an  exportable  CBT  setting. 

With  regard  to  resources,  this  group  would  have  liked  exemplar 
lesson  models  at  the  beginning  of  the  project,  but  they  saw  no 
value  in  such  models  now.  They  were  not  using  the  Hill  AFB 
style  guide  because  they  believed  that  it  was  not  meaningful. 
They  cited  the  need  for  a  centralized  repository  of  knowledge 
and  tools. 


V.  Cailc3,u^.,ioji,s 

The  following  list  reflects  needs,  issues,  and  problem  areas 
that  we  believe  are  relevant  in  the  development  of  AIDA 

1.  There  is  a  general  lack  of  knowledge  about  educational 
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principles  and  implications  for  instructional  design  at  all 
levels.  Several  individuals  had  developed  instructional 
design  expertise  and  were  sensitive  to  such  things  as  working 
memory  overload  and  varying  requirements  for  simulation 
fidelity,  but  many  of  these  experts  would  soon  be  transferring 
back  to  field  organizations.  There  is  a  need  for  a 
centralized  source  of  instructional  design  expertise 
(principles,  standards,  tools,  examples,  etc.)  which  is 
accessible  to  instructors,  designers,  developers,  and 
managers . 

2 .  There  is  an  immediate  demand  and  need  for  tools  to 
automate  the  ISD  documentation  process  and  develop  other 
course  materials.  Such  tools,  should  they  be  developed, 
should  allow  users  to  focus  on  developing  and  revising  course 
materials,  making  ISD  documentation  a  background/hidden 
support  system. 

3.  There  is  very  little  understanding  of  alternative 
instructional  models.  Instructor-based  training  is  fairly 
well  understood,  although  the  instructor's  role  as  a  model  for 
a  professional  military  technician  is  not  we 11 -documented  in 
the  ISD  process  for  instructor-based  training.  There  is  a 
definite  and  obvious  need  for  well-articulated  guidelines  for 
the  instructor's  role  in  a  CBT  setting. 

4.  Decisions  which  affect  instructional  design  are  made  at  a 
variety  of  levels,  including  acquisition  groups,  using  groups, 
ATC  headquarters,  TMs,  TDBs,  and  the  training  specialists  who 
design,  develop,  deliver,  and  maintain  the  instruction.  Some 
decisions  made  by  cne  group  may  seem  uninformed  and  capricious 
to  another  group.  Consistently  effective  decision-making  in 
the  instructional  design  process  is  needed.  Instructional 
design/decision  aids  which  offer  informed  alternatives  and  can 
produce  justifications  would  be  useful  in  working  towards  a 
consistently  effective  decision-making  process. 

5.  Many  of  those  involved  at  all  levels  in  instructional 
design  and  development  felt  those  involved  at  higher  levels 
did  not  fully  support  or  understand  their  specific  tasks. 
Decision  aids  should  be  aimed  at  both  informed  and  uninformed 
audiences  and  should  provide  justifications  and  alternatives. 
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